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Abstract 

In 1998, a state university received grant funding to convert their Special Education Blindness and 
Visual Impairment graduate degree program to an online format. At that time, commercial web course 
management systems were not accessible to blind and visually impaired users. As a result, grant designers 
developed a custom, accessible platform, which led to accessibility standards for online courses and to an 
award-winning design and interface. In 2002, the university licensed Blackboard™ and encouraged the 
migration of all online delivered courses to this standardized system. After determining that the newer version 
met accessibility standards, the instructional design staff conducted a qualitative return on investment analysis 
to evaluate whether the migration to Blackboard™ would cause losses in instructional and interface quality. 
This paper explores the pro cess for developing a qualitative return on investment and how the benefits and 
tradeoffs were analyzed related to maintaining an internally developed system versus migrating to 
Blackboard™. 



Introduction 

Traditional methods for analyzing whether a decision is ultimately a good decision have focused on 
measures that can be quantified and that ultimately contribute to a financial bottom line. However, in 
environments that may not be driven by financial bottom lines - educational settings, non-profit organizations 
or grant activities within a higher education institution - such methods for analyzing an important decision fail 
to capture the real variables in the decision. Furthermore, increasing demand for attention to assessing social 
impact of decisions (Barbour, 1993; Kaufman, 2000) is driving the need for newer methods that take into 
consideration a broader array of variables and the ultimate impact of a decision. 

The ability of return on investment (ROI) and cost-benefit analysis (CBA) to accurately and fully 
analyze the impact of a decision is being called into question. Barbour (1993) explains that ROIs, CBAs and 
risk assessments are limited because they often leave the real benefits or dangers unassessed since those are 
qualitative aspects of a project that cannot be quantified. Often those unassessed benefits or dangers are 
impacts upon human lives or the environment. In response, agencies such as the Office of Technology 
Assessment and the United Nations Development Program have developed mixed- method analysis procedures, 
such as the “Human Development Index” (Barbour, 1993, p. 53), that analyze both the quantitative and 
qualitative factors of decisions or policies. In business and industry, Kaufman (2000) has proposed an 
Organizational Elements Model as a tool companies can use to assess their ultimate benefit to and impact upon 
society. 

While every decision may not be an earth-shaking one requiring analysis of societal good, there are 
many instances where qualitative aspects of a project must be assessed and analyzed in order to determine the 
real costs and benefits. The impact of a decision upon employee attitudes, public perception of quality, and 
even changes it causes in processes or specific design standards are all examples of more qualitative variables 
that may be involved in a decision. This paper explores a specific instance where a qualitative ‘ROI’ process 
was developed in order to assess a decision about migrating online courses from one platform to another. While 
some aspects of the migration issue could be quantified, many could not. Still, analysis and data backing the 
decision were needed by management. We will describe the context of the project and discuss why a qualitative 
ROI was appropriate. We will also define ROI, the questions we investigated, the methodology developed to 
conduct the analysis, and the findings the analysis yielded. 

Qualitative ROI Project 

When a regional funding organization first awarded a state university’s Blindness and Visual 
Impairment Program grant funding in 1998 to convert its Master’s degree program to an online format; off-the- 
shelf, commercial web course management systems were not accessible to blind and visually impaired users. 
Because 10% of the students enrolled in such a program had visual impairments and one faculty member was 
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blind, it was imperative that the grant team develop a custom web course management system and identify 
online synchronous and asynchronous tools that were accessible. Over the length of three years, 15 courses 
were developed on this custom, internally -developed platform, and a virtual campus web interface was 
developed to support distance students. 

Based upon the success of the program, in 2001 the university was awarded a second federal grant, 
which significantly increased the scope of the project. It provided the necessary funding to continue the online 
program in Blindness and Visual Impairment and to convert two other programs, the Deafness and Hard of 
Hearing and Severe Disabilities Master’s degrees to an online format. This federal funding was also used to 
create a national center related to disability services and education. The online Master’s degrees are now a part 
of the center’s expanded teacher training function. Additionally, the center contracts with other universities to 
support the conversion of their low-incidence disability degree programs to an online format. Clearly, the 
quality of the online courses and programs - both in terms of instructional design and accessibility - formed a 
cornerstone of the center’s work. 

At the same time the center received this federal funding and expanded its efforts, the university in 
which the center is housed licensed Blackboard™, a commercial web course management system, which is 
maintained and administered by the university’s faculty development center. During the fall of 2001, the 
center’s staff members conducted a research study to determine the practical accessibility of the product. The 
results showed that the majority of the Blackboard™ interface met accessibility standards (Conn & Ektermanis, 
2001 ). 

The federal funding impacted the size and structure of the instructional design team. Three additional 
instructional designers were hired to support the expanded missions of the center. One challenge was to 
maximize the impact of the new instructional design team members. Even with an increase in staff members, it 
was difficult to address the issues of limited faculty control that were an inherent part of the internally- 
developed system and the increased workload of maintaining courses in the Blind and Visual Impairment 
program as well as the extensive work needed to convert the Deafness and Hard of Hearing and Severe 
Disabilities programs. 

The new instructional design staff members brought varying degrees of technical expertise thus 
making it necessary to consider a migration to a commercial web course management system with a graphical 
user interface. Given the results of the accessibility research study and the changes in size and structure of the 
instructional design staff, it was determined to be an appropriate time to evaluate the benefits and tradeoffs 
related to maintaining the internally -developed system versus migrating to Blackboard™. Once the project was 
determined to be appropriate and necessary, the instructional design staff conducted a review of methods to 
determine an appropriate process for conducting this analysis. 

Literature Review 

What is ROI? 

ROI is an acronym for return on investment. It is a method for measuring the worth of an investment 
and has been primarily utilized for businesses purposes. In the 1990s, the use of ROI for calculating the value 
of training and performance solutions began to be addressed by the human resource development and 
performance improvement fields (Phillips, 1997). In a human resource development context, “ROI practices 
are a means of economically connecting the performance goals of efficiency and effectiveness with selected 
interventions and performance results” (Swanson, 1999). The literature base for these fields advocates using 
ROI as a means of measuring, documenting, and communicating the value of support interventions to both 
justify projects as well as to build cases for continued or new funding (Pine & Tingley, 1993; Phillips, 1997; 
Stolovitch, 2002). 

Connecting ROI to Kirkpatrick’s Evaluation Model 

ROI has been connected to Kirkpatrick’s (1998) evaluation model. Kirkpatrick’s original model 
included four levels: 1) Training Reaction, 2) Learning, 3) Behavior, and 4) Business Results. Training 
Reaction is often gathered through end of training or course evaluations and captures data related to participant 
satisfaction and comments related to how the training or education may transfer to work situations. Learning 
evaluation data attempts to capture participant perception of their achievement of objectives related to 
knowledge, skills, and attitudes. Evaluations of behavior, also referred to as Application, investigate changes in 
work performance. Business Results evaluates the impact of the interventions on related business variables. 

Phillips and Phillips (2003) added two new levels to Kirkpatrick's model placing “ROI” and 
“Intangible” at the fifth and sixth levels, respectively. As mentioned earlier, ROI is a process for measuring the 
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costs and benefits of an intervention. Intangible is the documenting and reporting of relevant variables that are 
not easily converted to a monetary value. Although the levels of Business Results and ROI may appear similar, 
they differ in that Business Results attempts to measure changes in the business related to the intervention, such 
as productivity or attitudes of employees or profitability levels. ROI, on the other hand, focuses on comparing 
these identified benefits (or disadvantages) of the intervention with the costs of implementing the intervention. 

Types of ROI Evaluations 

There are several different types of ROI evaluations that can be conducted. Phillips, Stone, and 
Phillips (2001) directly align the ROI evaluation options with Kirkpatrick’s original model, which forms a 
framework for the timing of data collection as well as a consideration of the levels of credibility, accuracy, cost 
to implement, and difficulty to implement. For example, when conducting an ROI measure related to 
Kirkpatrick’s first level of evaluation. Reaction, data is collected during and/or at the end of the training. 
Credibility and accuracy of these measures tend to be lower since transfer of training to work settings has not 
yet occurred, but these evaluation measures are often less expensive and difficult to implement. Collecting 
evaluation data related to Business Results, Kirkpatrick’s fourth level, can be very credible and accurate since 
the intervention will likely be implemented by this point; however, these types of measures are typically more 
expensive and difficult to collect. 

In addition to aligning ROI evaluations with Kirkpatrick’s levels, Phillips, Stone, and Phillips (2001) 
include one more option — Forecasted ROI. A Forecasted ROI, also referred to as worth analysis (Stolovitch, 
2002) and anticipated ROI (Parkman, 2002), is conducted before an intervention is implemented and is the type 
of ROI employed in this study. It can help provide justification for a project as well as provide baseline data 
that can be compared with post-project results. Forecasted ROIs are based on estimations and therefore may be 
less credible or accurate than ROIs calculated on post-project data. However, they have the benefit of being 
inexpensive to develop and less difficult to conduct. 

Sequence and Criteria for Conducting ROI 

Before conducting a ROI, a front-end analysis should be conducted to identify “what the desired 
business state should be, what the current or actual state is and then [to] characterize the gap between the two 
states in terms of magnitude, value and urgency’’ (Stolovitch, 2002). This front-end analysis provides data for 
determining an appropriate solution and clarifying project goals in terms of business results. The results of the 
ROI can also be used for developing project evaluations and comparing the benefits of the solutions to the 
potential costs (Parkman, 2002). 

Phillips (1997) describes ten criteria for conducting a traditional ROI. These criteria form a set of 
guidelines to follow when investigating the ROI of a project, product, or training. These guidelines can be 
summarized as 1) keep the process simple by employing practical, feasible methodologies; 2) design an 
economical process that is easy to implement, has the potential of becoming routine, and can be applied to a 
various types projects as well as to both pre -project and post-project data; 3) choose evaluation techniques or 
research methodologies that are credible, theoretically sound, and based on accepted practices; and 4) create a 
process that can utilize all types of data and include the costs of the program. 

Qualitative ROI 

Financial factors are often not the only variables that need to be considered when gathering data to 
estimate or judge the value of an intervention. Intangibles are variables that are critical to the overall project or 
solution but are not easily converted to monetary values. As mentioned earlier, Phillips and Phillips (2003) add 
‘Intangibles’ as a sixth level to Kirkpatrick’s original model. Swanson (1999) states “criteria other than ROI are 
being used to gain support for performance improvement programs. Although there appears a difference of 
opinion in the literature regarding whether ‘Intangibles’ are or are not a ROI measure, we chose to adapt 
procedures to create a qualitative ROI process given the mix of data sources available and the context of our 
study. 

Swanson (1999) describes several qualitative factors that should be considered. 

1. Appropriateness of the program to the organizational culture and tradition 

2. Availability of the program 

3. Perceived quality of the program design 

(p. 836) 

These are especially important to take into consideration when working with non-profit organizations or 
educational institutions where economics may not be the key driver and where hard program costs may be 
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difficult to access or are not directly financed by the project’s budget. This was the context for this particular 
proposed evaluation project. 



Purpose of the Qualitative ROI Project 

A front-end analysis had been conducted to verify that a course management system was (still) needed 
to deliver the three low incidence disabilities graduate degree programs and to engage in contract work with 
other universities. Additionally, the version of Blackboard™ licensed by the university had been thoroughly 
tested to ensure it was accessible and met Section 508 standards. Given the results of this up front analysis, the 
question that remained was what would be the benefits and tradeoffs of migrating courses to Blackboard™ 
versus continuing to use the internally developed web-based course management system. From this key 
research question, the following secondary questions were developed. Would Blackboard™ (or policies related 
to Blackboard™): 

1) Decrease course development time for the instructional design staff, so that more time could be 
devoted to the other missions? 

2) Allow the center to continue to deliver high quality courses with cutting edge designs? 

3) Allow the center to maintain and contribute to quality instruction in the areas of accessibility, 
increased features, and increased control for instructors? 

4) Increase the center’s return on monthly fees being paid to the university’s information technology 
and faculty development departments? 

5) Enhance campus relationships between the faculty development department and the center, and 
add value to the university? 

6) Support the center’s ability to partner with other institutions for delivering courses? 

In addition to exploring the questions listed above, the center’s instructional design staff also felt the results of 
this study could prove valuable to other university special education departments with whom the center 
consulted and who were considering whether to develop a course management system internally or use an off- 
the-shelf product. 

Methodology 

Going into the project, the evaluation team realized there would likely be many intangible variables 
and many other variables that would be difficult to quantify, given the fact that departments within institutions 
of higher education typically do not charge internal clients for the services they provide and often institutional- 
wide site licenses are purchased for software. Given this context and the potential variables that would be part 
of the overall analysis, a methodology that incorporated the ROI criteria and qualitative research techniques was 
employed. Qualitative inquiry methods are an appropriate approach for descriptive studies and for researching 
practical problems (Creswell, 1998; Merriam, 1988). Given the descriptive nature of qualitative studies, the 
findings or results include detailed narratives regarding questions being researched. These descriptions are 
intended to paint a picture for the reader of the situation or entity studied. This is done through the use of text 
and images as well as through quotes, examples, or other appropriate artifacts (Wilson, 1979). 

Project team 

As with many return on investment projects, a team was formed to conduct this study. In qualitative 
research it is important to inform readers of the biases the researcher or researchers bring to the project. 
Informing readers of researcher bias allow them to draw their own conclusions regarding the trustworthiness of 
the findings. The research team for this project consisted of two instructional design center staff members who 
were also pursuing doctorate degrees in Educational Technology. One of the center’s instructional design staff 
members proposed the project and leaned towards migrating to Blackboard™. The second center instructional 
design staff member had been with the project since the receipt of the first grant and this staff member’s work 
was central to the creation of the internally developed course management system interface; she was hesitant 
towards the idea of migrating to Blackboard™. In addition, the project team included five other instructional 
designers with varying levels of expertise. These consultants had no association with the center. 

Data collection 

Data collection for qualitative studies often involves multiple sources (Creswell, 1998; Merriam, 

1998). These sources can include “documents, archival records, interviews, observation, [or] physical artifacts” 
(Creswell, 1998, p. 65). Merriam (1998) states, “interviewing is probably the most common form of data 
collection in qualitative studies in education. In numerous studies it is the only source of data” (p. 70). Using 
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more than one method for collecting data and verifying emerging themes reinforces the results of qualitative 
studies. This practice of collecting and analyzing a variety of data sources is called data triangulation (Denzin, 
1978). 

The data collection for this study involved multiple sources. We interviewed the center’s director and 
conducted two separate interviews with staff members from the faculty development department. We collected 
documentation including a slide presentation prepared by the center’s technology manager, which explored 
technical implications of using various combinations of servers and courseware for delivering courses online 
and email correspondence with the Blackboard™ staff member responsible for accessibility issues. We also 
accessed two websites, the Blackboard™ company website that discusses the accessibility of the tool and the 
Section 508 website, a federal government site dedicated to the implementation of federal legislation for 
accessibility of multimedia information. Finally, we utilized the results of the Conn and Ektermanis (2001) 
study that had been conducted to investigate the practical accessibility of the Blackboard™ interface. 

Data analysis 

The interview data were analyzed using qualitative coding techniques. A characteristic of qualitative 
research, as defined by Merriam (1988), is inductive reasoning. Inductive reasoning refers to the emergence of 
concepts and themes through data analysis. The researcher may begin the data analysis with an outline of 
possible concepts or themes he or she expects to find, but these initial codes are often revised, eliminated, or 
added to through the coding process. 

The detailed interview notes were transcribed and were read and re-read by the project team. The 
transcriptions were then analyzed using basic qualitative analysis methods. The basic qualitative analysis 
method followed by the team included first-level coding, which involves how one differentiates and combines 
the data the researcher has retrieved and the reflections one makes about the information. These codes are 
designed to be descriptive labels for identifying chunks of information (Miles & Huberman, 1994). The 
research questions were used to guide the analysis of the data, and were the basis for the initial list of codes 
used to analyze the interview transcriptions. A content analysis of the documentation collected was also 
conducted using a first-level coding method to triangulate the data collected through the interviews. 

For first- level coding, the team wrote all the data on large pieces of paper that were then posted on the 
walls of the room. One member of the team typed these pieces into an electronic format as the group worked. 
Once all the data was posted on the wall, the group started by simply numbering each piece of data. The first 
data piece listed received number one. If the next piece was similar to something already numbered, it received 
that same number. Otherwise, a new number was introduced. In the first round of coding, six categories of data 
emerged. The person entering it electronically reorganized the pieces into those six categories. 

The next step in the analysis process was second-level pattern coding. Second-level pattern coding is a 
method for grouping first-level codes. Pattern coding is used to identify emergent themes or explanations. The 
primary purpose of second-level pattern coding is to assist in getting to the next level of analysis - beyond 
simple description (Miles & Huberman, 1994). 

For second-level coding, the team printed out the initial six categories and placed these on the wall. 
They then discussed what the categories would be called and whether categories should be maintained or 
whether some overlap still existed between categories. After further analysis, the project team reduced the 
categories to three core themes: Quality, Time and Cost. Once the first-level coding and second-level pattern 
coding was complete, the project team analyzed the results and synthesized data into a descriptive report 
delivered to the center’s management team that included quotes, examples, and images to convey the findings 
and recommendations of the study. 



Findings 

Based on our analysis, the issues, concerns and solutions collected from all sources were grouped into 
three key themes: Quality, Time and Cost. Figure 1 visually depicts the relationship that emerged between 
these three themes. The findings of this study indicate that the main considerations short-term were the issues 
of the loss of the center’s identity and ‘sense of place,’ and the impact on instructional designers’ time. The 
findings also pointed to concerns related to long-term sustainability of the online courses as well as different 
roles and time investments for the center’s staff. In addition, benefits of the migration to Blackboard™ 
emerged as well as several specific issues that were documented as Recommendations. Table 1 contains a 
summary of the findings categorized as ‘costs’ and ‘benefits.’ 
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Figure 7. Visual representation of the relationship between the three major themes emerging from 



Table 1. Summary of Findings Categorized as Costs and Benefits 





[Variable 


Costs 


Benefit 


Quality 


Accessibility of 
Blackboard™ 


No longer an issue except for 
minor problems with the interface 
and the chat room tool 


With increased collaboration 
between the center staff and the 
faculty development department 
the center’s accessible chat room 
tool could be made available to the 
entire university 


Intetface Design 


Would lose ‘sense of place’ and 
community designs 




Instructor control 




Instructors would have more direct 
control over making changes in 
their courses; a feature that was 
not available in the internally 
developed system 


Instructional 
Design Quality 


Loss of infrastructure that 
supported webs of information and 
data pieces 




Technical Issues 


University server less stable, more 
down time 


More portability across programs 
and universities 


Development 

Support 




Long-term, university funded 
support for course development 
and maintenance 


Time 


Course 

Development 




Decrease course development time 
for the center’s staff since the 
faculty development department 
could assist with course 
development 


Collaboration 


The center’s staff would likely 
need to spend more time 
participating in faculty and staff 
development trainings to address 
issues of accessibility 


This would allow the center to add 
value to the university community 
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Help Desk 
Support 


Student confusion surrounding the 
accurate logon and password for a 
specific course 


With the migration to one system 
for the university, the students 
would more likely become 
accustomed to the appropriate 
logon and password; additionally, 
the university help desk would be 
able to assist all students, giving 
students one central place to 
contact or for instructors to refer 
students to 


Cost 


Multimedia 

Development 




Faculty development department 
provides (free) audio, video and 
graphic development services 


Information 
Technology Fees 


The center was required to pay 
$40 per credit hour to the 
university’s information 
technology department for support 
services, but the center was not 
receiving any value for these fees 
since all course development and 
support to students and instructors 
was provided internally 






Risk Factors 


Unknown outcome of annual 
contract renegotiations between 
the university and Blackboard™ 
which could potentially require 
distribution of Blackboard™ 
product fees to the department 





Quality 

The migration of courses to the university’s web course management system. Blackboard™, raised 
several concerns related to the quality of current courses, specifically issues of handicapped accessibility and 
good instructional and visual design. Based on the data analysis, it was determined that these concerns could be 
addressed through proposed solutions or balanced by gains from the proposed migration. 

Accessibility: Earlier accessibility issues related to the Blackboard™ interface had been addressed, and 
most course components now met federal accessibility guidelines under Section 508 which stipulates that 
electronic and information technology should be programmed in such a way that individuals with disabilities 
can access and use the information and data in a way that is comparable to the access and use by individuals 
without disabilities. Some features of the Blackboard™ interface were still not accessible, though, such as the 
chat rooms. Interviews conducted for this ROI revealed that university staff were willing to allow the center to 
link in its own custom, accessible chat rooms and other tools and even make those tools available to users 
across campus, adding further benefit to the entire campus. 

Interface Design: The custom, internally-developed interface featured an identity and ‘sense of place’ 
that was designed to be extremely user friendly. The original interface created a sense of community by 
developing a virtual campus around the online courses and programs, and an infrastructure that allowed students 
to connect with each other outside of class or with outside experts for informal discussions, much like a 
physical university center would host social and informal events. Students could also access ‘offices’ and 
‘buildings’ that they needed to be successful in their studies, such as financial aid, the library and faculty 
offices. This instructional strategy of community was supported with visuals and identifiers that all created a 
‘sense of place’ where the students felt like they were a part of a program and a university, not just taking online 
courses. Within classes, a visual interface resembling a classroom had been developed that included pictures of 
faculty and other features that helped students adapt more quickly to this new innovation of online learning. All 
this work had been created based on research on learning communities (Wenger, 1998; Palloff & Pratt, 1999) 
and change facilitation (Rogers, 1995; Hall & Hord, 2000). 

A concern that surfaced during this study was that this distinctiveness would likely be lost with the 
move to Blackboard™. However, benefits to be gained with the migration appeared to offset this concern. For 
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example. Blackboard™ would allow faculty more control over course changes and updates as well as a wider 
array of course tools and features. According to the literature base on change, such a form of empowerment 
would be critical for faculty adoption of the innovation because it fosters buy-in and improves success 
(Ellsworth, 1995) and allows the stakeholders to participate in the very technology that will impact their work 
(Ellsworth, 1997; Ely, 1990). A final benefit of offsetting the concern over loss of distinctiveness was that by 
using the university’s official contracted system, long-term support for course development and maintenance 
was assured, if funding would not be available to support center instructional design staff in the future. 

Instructional Design: In addition to accessibility and interface design concerns, were concerns of 
instructional design quality. Courses had been redesigned for online learning based on cognitive apprenticeship 
and situated cognition principles (Brown, Collins, & Duguid, 1989; Herrington & Oliver, 2000). Instructional 
designers had built scavenger hunts using the server technology, created information databases students 
collaboratively populated as course projects, and even built a set of scaffolded case studies and support tools 
that all made extensive use of the custom, internally -developed platform. In addition to the concern as to 
whether these quality learning experiences could be preserved using the Blackboard platform, was the practical 
consideration that migration of this custom content would not be straightforward, but would require time and 
expertise to ensure a well-managed process. However, as with the accessibility issues, the qualitative ROI 
revealed that the center could maintain complex structures on its own server and link to those from within the 
courses on the Blackboard™ server. Thus, the center could capitalize on the benefits of Blackboard™, but 
maintain the past work and future flexibility that had contributed to its reputation for quality. 

Technical issues: Additional quality concerns centered on technical issues. The university’s 
information technology server was viewed as less stable than the server running the custom, internally- 
developed course management system. The university’s information technology server had been down for two 
months total out of the year when the center’s server was not, one of those downtimes occurring during the 
critical time when courses started. Furthermore, the listserv functions within Blackboard™ or from information 
technology were either not as reliable or did not provide the same capabilities as the original server structure. 
However, the center would be able to institute its own policy and control related to backing up courses. 

Portability: One final advantage of migrating the center’s online courses to Blackboard™ included 
cross -institutional portability. By using a common system, other universities’ special education departments 
could partner with the center to develop courses. While a custom solution provided the center a high degree of 
strength and flexibility, other programs did not have the resources to maintain something similar. Their need to 
be on a standardized, university-maintained system outweighed the advantage of the internally developed, 
custom solution. 

Time 

One of the major issues we investigated was whether switching to Blackboard™ would decrease 
course development time for the center’s staff. Through this process, we discovered that one of the services the 
faculty development department offers (free of charge) is to do course development for faculty members. To 
date, mo st faculty members had opted to develop their own courses, so this service was not being widely used. 
Other issues that surfaced related to time were the requests by the faculty development department for help with 
training their staff on development techniques related to accessibility as well as assisting with their faculty 
development courses on issues related to accessibility. 

Based on our analysis, it appeared that in the short-term switching to Blackboard™ might increase 
work for the center’s staff to meet the requests made by the faculty development department. Long-term, the 
data appeared to support the objective of decreasing course development time for the center’s staff. Although 
these results did not support decreased staff workload in the short-term, the migration would likely enhance 
campus relationships between the faculty development department and the center, and add value to the 
university. 

The final issue related to time that surfaced was the logon/password confusion encountered by students 
at the beginning of each semester. This was an issue that the faculty development department, the center, and 
Special Education instructors currently dealt with each semester regardless of the web course management 
system. The center’s internally developed system required unique course passwords that were reset each time 
the course was taught. By migrating to Blackboard™, students would be able to use their university logon and 
password. Additionally, the university help desk would be able to respond to requests for help regarding the 
logon and password, providing another source of help for students beyond the center’s staff members and the 
course instructor. 
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Cost 



Cost is an inevitable issue when considering the use of new tools and services. Upon analysis, it was 
found that the data pertaining to cost could be organized into three categories or cost issues for this evaluation 
project, Free Services, Return on Information Technology Fees, and Ris k. The data regarding Free Services 
revealed the faculty development department’s willingness to offer free audio, video and graphic development 
to supplement the instruction. These services would sustain the center’s goal for providing students with 
authentic and innovative learning environments through the utilization of technologically advanced tools. 

The factors fitting under the Return on Information Technology Fees exposed a current lack of return 
for fees paid by the center to the university’s information technology department. The center was required to 
allot $40 per credit hour to the university’s information technology department for support services. At the time 
this qualitative return on investment analysis was conducted, the center had not been able to offer the 
university’s information technology department support to students and faculty since the center staff maintained 
the custom course management system. Migration to Blackboard™ would provide the opportunity for the 
center to take advantage of the support services offered by the information technology department, thus creating 
a better return on investment for the fees paid by the center. 

Based on the results of the analysis of cost data, the outcome appeared to be in favor of a migration to 
Blackboard™, with the exception of the Risk factors. Risk factors associated with cost related to the unknown 
future outcomes of contract renegotiations between Blackboard™ and the university. Renegotiations occurred 
annually, and terms of a new contract could at some point include a dispersal of cost onto the departments and 
organizations that use Blackboard™. 

Recommendations 

Throughout the data analysis process, ‘action items’ continued to emerge that would need to be 
addressed should the center decide to migrate to Blackboard™ for online course delivery. The evaluation 
project team brainstormed solutions that could be implemented to address the ‘costs’ (e.g., concerns or 
disadvantages) highlighted in the findings. Presented in Table 2 is a summaiy of the issues that emerged from 
the data as well as recommended solutions. 

Table 2. Summaiy of Issues Related to Migrating to Blackboard ™ and Recommended Solutions 



Issue 


Recommended Solution 


Loss of identity 


■ Providing a link from the main Blackboard™page to the center information, 
stating ‘supported by the center’ in some fashion - (create a general page for all 
courses) 

■ Put the center’s ‘design element’ on Blackboard™ page 

■ Support accessibility through training faculty development department staff and 
instructors 


No sense of 
community 


■ With every center course, create a link for all students to access the center’s 
community (create a general page for all courses) 

■ Encourage faculty to develop community through course design 

■ Use orientation for students at beginning of course to inform them of the center’s 
virtual campus 


Managing the 
load 


■ Migration Plan 

Incorporate change management into the plan 
=> Limit the number of transitions and course management interfaces for 
students and instructors 


Policies 


■ Negotiate policies related to: 

=> Courses being taught for other universities 

Continuing support for online courses and migration to any new system(s) if 
the university drops the contract with Blackboard™ 


Relationships 


■ Establish and foster clear, positive communication lines 
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■ Support accessibility through trainings sponsored by the faculty development 
department and through individual consultation with instructors 

■ Extend accessibility support to library and web manager (for entire campus) 


Cutting edge 
design 


■ Not compromised by Blackboard™ if Educational Technology staff mediates 
between faculty and Blackboard™ 


Stability of 
servers/ longevity 
of Blackboard™ 
contract 


■ Establish systematic back-up procedure for all courses 



Summary of Findings 

Thus the findings showed that the migration to and use of Blackboard™ would not jeopardize quality 
of online courses and would likely increase quality given the additional features of the system. Over time, use 
of Blackboard™ would translate into more time for the center’s Educational Technology staff members to 
devote to broader instructional design efforts rather than course development issues. Under the university 
model and contract in place at the time, the migration to and use of Blackboard™ would incur no cost to the 
center and would actually increase return on monies already being paid to the faculty development department. 
Additionally, if sources of funding for Educational Technology staff were decreased or cut completely, the use 
of Blackboard™ would ensure longevity of courses beyond any particular grant. Major and minor issues 
related to Blackboard™ could be addressed through policies, procedures, or products as identified in the 
Recommendations section. Therefore, it was the recommendation of the project team that the center migrate its 
courses to the Blackboard™ system and implement the recommended solutions to make such migration as 
smooth and beneficial as possible. 



Conclusion 

Overall, the qualitative approach aligned well with the type of data the team was able to collect for this 
evaluation project. We were able to address the intangible issues related to delivering quality courses. Only a 
few hard financial costs were gathered as part of the data for this study; however, this qualitative approach 
could be easily combined with more traditional financial ROI calculations if it was appropriate for the project 
and the financial data was available to analyze. The final report did provide the center’s management team with 
detailed analysis of both the benefits and disadvantages, allowing them to make a data-based decision regarding 
whether or not to migrate to Blackboard™ or remain with the internally developed course management system. 
The center did choose to migrate to the Blackboard™ server and implemented many of the suggested 
recommendations for mitigating against lingering costs. 
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